In modem computer systems, the storage of data to certain memory areas is managed by a file system. This file system is often difficult or impossible to add to or modify directly, because the file system may be complex or proprietary, and often is both. In certain cases, however, it may be desirable to add storage-related functionality to a computer system. For example, where a file system does not establish and enforce limits on the usage of storage by individual users, adding functionality which enforces such limits may be desirable. As discussed, modifying the file system to add such functionality may be difficult or impossible to do.
In order to add storage-related functionality to a computer system, then, functionality may be added via a file system filter rather than by changing the file system. In order to create and use a filter, no change needs to be made to the file system. Rather, the filter sits on top of the file system, with data traffic to the file system passing through the filter. In this way, the filter can monitor the internal status of the file system and, if necessary, block, modify, and/or enhance certain traffic. For example, if a quota system is needed in which the use by a given user of the storage managed by the file system is limited, a quota filter can be installed which monitors data traffic to the file system, discerns which traffic concerns write commands issued to the file system, and uses this information in order to determine what the relevant usage is for different specified users. The quota filter thus generates and saves file system metadata (data regarding the state of the file system) in order to track usage of the file system.
In addition to monitoring traffic, in some cases the quota filter blocks a write request to the file system which requests storage of data which would cause a quota policy (e.g., a limitation on the amount of data which can be stored in a particular directory by a particular user) to be violated. If necessary, the quota filter also sends appropriate messages back to the requesting entity regarding the situation.
While a quota system has been described, other filters may be installed which implement additional functionality which is based on monitoring or intercepting traffic to the file system. For example, encryption/decryption may be implemented using a filter. A hierarchical storage management (HSM) filter or“ghosting” filter may be used in order to implement a file achieving in a way which is invisible to the user, with files being moved from the file system according to a policy and provided by the filter if subsequently requested. For example, files may be moved from the file system to alternate storage if they have not been used for 30 days, and then, if a file which has been moved is the subject of a file system request, the filter, upon receiving the request, restores the file from the alternate storage.
One problem which occurs when such monitoring or interception of traffic to the file system is being implemented by a filter, and the computer system in which the file system is implemented shuts down in a manner which is not a clean shutdown. For example, if the system is power cycled or crashes and is rebooted, the shutdown procedures may not have time to execute. Such a shutdown is known as a dirty shutdown. Many file systems include techniques which allow the file system to recover from such dirty shutdowns without a loss of file system integrity. While maintaining the integrity of the file system, these file system techniques may cause behaviors which cannot be accurately tracked by a filter, and thus may endanger the integrity of the filter's metadata.
For example, an extending write request is a request which extends the size of a preexisting file. Where extending writes have been started but not completed before a sudden shutdown, in some file systems, such writes are rolled back by the file system upon rebooting. When this occurs, writes which have been logged by the quota filter as having occurred have in actuality been rolled back by the file system. Because of this, a filter may contain incorrect information. For example, where a quota filter is being used to track a user's writes, if one of the user's writes has been rolled back, the quota filter's assessment of how much space has been used by the user may be different from the actual usage. Such discrepancies may be significant, and the discrepancy between the actual state of the file system and the information maintained by a filter such as a quota filter about the state of the file system may only grow with time and additional dirty shutdown events. This can cause significant problems with the proper functioning of the filter. For example, in the case of a quota filter enforcing a certain restriction on how much space a specific user can use in a specific directory, where the quota filter does not have an accurate understanding of how much space the user is actually using in that directory, the enforcement of the restriction will not function correctly. The user will either be allowed to store more information in the directory than the user should, or the user will be restricted from using more space in a directory even though the user has not used as much space as the user has been allotted in the directory.
One solution to this problem is for the filter to rebuild its metadata information regarding the file system by canvassing the file system to determine its current state. For example, when a quota filter is installed on an existing system, the quota filter must query the file system for the size of each file that is relevant to the quota policies enforced by the quota filter, as well as any other relevant information (e.g. which user is associated with files). This querying is done in order to determine what the current usage is of the file system. However, a quota filter will be most useful on a system where there are a lot of files stored in the file system and where there are a lot of users. It is precisely in these cases that rebuilding a quota filter's metadata will take a long time. In order to rebuild a quota filter's metadata by querying the file system regarding all of several hundred thousand files stored on a system, hours may be needed. This presents a difficulty in a system which, for whatever reason, undergoes numerous unexpected shutdowns. Additionally, the delay during the rebuilding, presents an additional problem—during such rebuilding access to users is either denied, which causes service interruptions which will likely be inconvenient at best and intolerable at worst, or access to users is allowed, and because, during the rebuilding, the quota filter is not fully operational, allowing access may cause other problems which the quota system was intended to remedy.